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§ ■ ABSTRACT 

In multiple hypothesis testing, the volume of data, defined as the number of replications 
per null times the total number of nulls, usually defines the amount of resource required. 
On the other hand, power is an important measure of performance for multiple testing. 
Due to practical constraints, the number of replications per null may not be large enough 
in terms of the difference between false and true nulls. For the case where the population 
fraction of false nulls is constant, we show that, as the difference between false and true nulls 
becomes increasingly subtle while the number of replications per null cannot increase fast 
enough, (1) in order to have enough chance that the data to be collected will yield some 
trustworthy findings, as measured by a conditional version of the positive false discovery rate 
(pFDR), the volume of data has to grow at a rate much faster than in the case where the 
number of replications per null can be large enough, and (2) in order to control the pFDR 
asymptotically, power has to decay to in a rate highly sensitive to rejection criterion and 
there is no asymptotically most powerful procedures among those that control the pFDR 
asymptotically at the same level. 
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Multiple hypothesis testing often faces situations where distributions under false nulls only 
have a subtle difference from those under true nulls. To make the difference evident, it 
is necessary to make repeated measurements from the distributions. As is well known, in 
order to attain a fixed power, the number of replications for each null, henceforth denoted 
by k, should be large enough. Roughly speaking, if the difference between the distributions 
under false nulls and those under true nulls is 5, then k should be of the same order as 
5~ 2 . However, in practice, oftentimes k cannot be large enough. There are many reasons for 
this: the time window that allows measurements is finite, the experimental unit associated 
with each null can only sustain a limited number of measurements, and so on. Under this 
circumstance, there are two important and related issues. 
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First, if the underlying objective of the tests is to identify even just a few false nulls 
irrespective of power, so that one can get useful clues for further study, then in some cases 
it may be reached by testing a large number of nulls. For example, suppose a population 
has a small fraction of "atypical" units. Then, in order to capture at least one of them, 
an approach is to obtain a large sample from the population, and, for each sampled unit, 
determine whether to reject the null that it is typical. In this case, even though at population 
level, the difference between atypical and typical units may be small, there is a chance that 
some atypical units will "show up" with pronounced differences from typical ones, making 
them easy to detect. In order to increase the chance, one would hope N, the number of 
examined units, to be as large as possible. Oftentimes, however, the amount of resources 
required for the tests is in proportion to iV or V = kN, the "volume" of data. This imposes 
a constraint on iV and V and raises the following question: provided that k cannot be large 
enough, what is the minimum value of N or V in order to have desirable test results? 

The answer to the question depends on what performance criterion to use for the tests. 
A useful criterion is pFDR < a, where a G (0, 1) is a pre-specified level and pFDR = 
E[R Q /R I R > 0] is known as the positiv e false discov ery rate, with R the number of rejected 
nulls and Rq th at of rejected true nulls 



E[Rq/(R V 1)] (IBeniamini fc Hochberg 



Storevl 2003 ). Comparing to the false discovery rate 



1995), the p FDR is more useful when the objective 



-r 

is to reject some nulls (lChj|2007d . IChi fc Tanll2008l ). In view of this, it is desirable to apply 
the idea of pFDR to data directly. Denote by X the data to be collected for the tests. Under 
a Bayesian framework, we propose the following variant of the pFDR criterion, 



P(inf E[R /R\R > 0, X] < a) > p, 



(1.1) 



where E[R /R \ R > 0, X] is defined to be 1 if R = 0, a, p G (0, 1) are pre-specified constants, 
and the infimum is taken over all applicable multiple testing procedures that are solely based 
on the data; cf. (12.21) . For any such procedure, once X is given, R is determined, whereas _R 
remains random, with its distribution determined by the posterior likelihoods of the truths 
of the nulls. This gives rise to the conditional expectation. As a result, the infimum in (11.11) 
is a function of X. The criterion means that, with probability no less than p, the data to be 
collected will yield one or more rejections, which, under a future study for verification, have 
an expected fraction of false rejections no more than a. The minimum value of iV or V will 
be studied under this criterion. 

The second issue is straightforward, that is, even when an arbitrarily large number of 
nulls can be tested, power may still be an important concern. Provided k cannot be large 
enough, how much power can be hoped for? 

The issue of minimum N oy V can be thought of as a dual to the issue that given N, 
how small the differences between false and true nulls c an be before it becomes virtually 
impossible to detect false nulls; see iDonoho fc Jinl (120041) and references therein. The issue 
of power is more extensively studied; see lEfronl (120071 ) and references therein. However, the 
setup here is different. First, both issues are considered in relation to k and it is necessary 
to take into account the fact that the distributions of test statistics not only depend on the 
underlying data generating distributions, but also on k, the number of replications from the 



2 



distributions for each null. Second, no sparsity is assumed for false nulls. Instead, false nulls 
are assumed to be increasingly similar to true nulls, while k cannot increase fast enough to 
compensate for this. As a result, the test statistics provide increasingly weak evidence to 
separate false nulls from true nulls. 

All our results are obtained for the case where distributions under false nulls and those 
under true nulls are known and the population fraction of false nulls is known as well. In 
practice, especially in exploratory studies, while there may be relatively good knowledge 
about distributions under true nulls, oftentimes there is little knowledge about distributions 
under false nulls or the population fraction of false nulls. Thus, the case we consider is an 
ideal one and the results provide limits on what can be achieved in more realistic cases. 

Section [2] covers preliminaries and identifies the quantity central to the analysis. Main 
results are stated in Section [3j Section 13.11 considers the asymptotics of the minimum N 
and V for general parametric models. Using Cramer-type large deviations, the results are 
established for the case where the growth of k is just a little slower than 5~ 2 . It shows that 
in this case, the minimum N and V have to grow much faster than in the case where k 
is of the same order as 5~ 2 . Sections 13.21 and 13.31 obtain refined results for tests on mean 
values of normal distributions with known variance and scaled gamma distributions. Section 
13.41 considers the power of multiple tests as k can not increase as rapidly as 5~ 2 as 5 — > 0. 
It shows that for procedures that asymptotically control the pFDR at a given level, the 
power decreases to and is highly sensitive to small changes in rejection criterion, and 
consequently, there is no asymptotically most powerful procedure among such procedures. 
Section H] concludes the article with a summary and remarks. The proofs of the main results 
are collected in the Appendix. 



2 Preliminaries 



Denote by Hi, . . . , H^ the nulls. Suppose that for each Hi, a sample Xn, . . . , X ik is collected. 
Let rji = 1 {Hi is false}. The analysis will be under the following random effects model: 



a, 



given rji 
given rji 



. . ,Xik), i>l, are i.i.d. such that 
F(i]i = 1) = a and 

0, Xn, ...,X ik are i.i.d. ~ f (x), 

1, Xn, ...,X ik are i.i.d. ~ f a (x), 



(2.1) 



where < a < 1 is the population fraction of fa lse nulls, and f n , f a are probability densi- 
ties u nder true nulls and false nulls, respectively (jEfron et al.ll200ll . iGenovese &: Wasserman 
20021 ). In this article, we will o nly consider the case where a is fixed. 

Following IChi &: Tan! (120081 ). a multiple testing procedure is a deterministic mapping 



d(X) = (d 1 (X),...,d N (X)), (2.2) 
such that Hi is rejected if and only if di(X) = 1. Under the random effects model, X = {X k i\ 
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and it can be shown that the criterion (11. ip can be rewritten as 

P(at least one £i{k) occurs, i = 1, . . . , N) > p, (2.3) 

' UX i5 ) (l-a)(l 



where := {X a , . . . , X lk satisfy FT J -fg*l > l± " n± ; (2.4) 

see Section @] for a sketch of proof. 

For fixed k, the events £i(k) are independent of each other and have the same probability, 
which depends on both k and the difference between f and f a . Suppose the difference can 
be parameterized by 5. For example, if fo = N(9q, 1) and f a = N(9, 1), then 5 can be taken 
as 9 — 9q. Denote the common probability of £i(k) by 

p k)S {a) = F{£i{k)), i = l,...,N. (2.5) 

Then (I2.3P is equivalent to 1 — (1 — Pk,s( a )) > p, or N > ln(l — p)j ln(l — Pk,s{ a ))- 

The case we will focus on is where the false nulls are increasingly similar to true nulls 
while k cannot increase fast enough to compensate for the decreasing <5; more specifically, 
<5 — > while k — > oo at a slower rate than 5~ 2 . Then, as maybe expected, pk,s{<y) — > 0. 
Provided that the growth of k makes sure Pk t g(ac) > 0, the minimum number of nulls and 
volume of data to satisfy (11.11) are 

ft = M h ' V. = kN. = £±4$** -!-, (2.6) 

Pk,5[OL) 1-P PkA a > l ~P 

respectively. Thus, the main task of the analysis is to find the asymptotic of Pk,s{ot). Note 
that under the random effects model, 

p k ,s{a) = (1 - a)¥ (£ t (k)) + a¥ a (£ t (k)), (2.7) 

where P is the probability measure under / and P a that under f a . 

Finally, some comments on the random effects model. It may be desirable to relax 
the assumption that the data distributions under false nulls are identical. To do this, one 
approach is to assume that under false nulls, the data obeys another random effects model, 
such that given rji = 1, a parameter 9j is first drawn from a dist ribution G, and then 



Xn, . . . , X ik are drawn from f e . ^ f (IGenovese fc Wassermanj|2002l ). However, by letting 



fa = J fe dG(9), it is seen that this model can be treated in the same way as (12. II) . Another 
approach is to use Poisson approximation to evaluate the probability in ( 12.31) . which does not 
require the di stributions under fa lse nulls be identical, and can even allow weak dependency 
between X^ ( Arratia et al. 1990h . However, a full development of the Poisson approach is 



beyond the scope of the paper. 

3 Statement of main results 

Recall that a is assumed to be fixed. Henceforth, denote Q a = (1/a— l)(l/a — 1). 
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3.1 Data volume for general multiple tests 

Suppose the observations take values in a Euclidean space Q and both fo and f a belong 
to a parametric family of densities {fg, 9 G 0} with respect to the Lebesgue measure dx on 
Q, where is an open set in M d . Suppose / = fg and f a = fg. Let £(9,x) = hif$(x). 
Suppose that for each x G fl, £(9, x) is twice different iable with 



,x) 



d£{9, x) 



d£(9,x) 



T 



£{9, x) 



d 2 £(9,x) 
d9 k d9 t 



We assume that {fg} satisfies regular conditions so that 

Eg[£(9,X)] = 0, Vaig[£(9,X)] = -E e [£(9,X)] = 1(9). 



where 1(9) is the Fisher information and Eg and Varg denote the expectation and variance 
under fg, respectively. For each 9, 9' G 0, by Taylor expansion, 



(9' -9Y£(9,x)(9' -9) 



o(\9> 



A(9,9',x)(9' -9) 



\x)-£(9,x) =£(9,x) T (9' -9) + 
= £(9,x) T (9' ~9) + 

where A(9,9',x) is a d x d symmetric matrix. Under regular conditions, one would expect 
that as 9' -> 9, E e [A(9,9' , X)} -> E e [£(9,X)} = -1(9). However, for the analysis, a few 
stronger assumptions are needed. 

Assumptions. 

1. I(9q) is positive definite. 

2. There are C > and r > 0, such that \I(9) - I(9 )\ < C\9 - 9 \ if \9 - 9 \ < r. 

3. M 3 :=sup ee0 £,|i(#,X)| 3 <oo. 

4. For any e > 0, there are positive numbers r, A and A, such that for any 9 and 9' G 
with \9 — 9'\ < r, if Xi,X 2 , . . . are i.i.d. ~ fg, then 



P 



k 

-Y,A(9,9',X i )+I(9) 



i=l 



> e 



for all k > 1, where for any matrix M 



m 



IMI 



max \rrij 



(3.1) 



Assumption 1 is standard. Assumption 2 holds if 1(9) is different iable at 9q, which along 
with Assumption 3 is satisfied by many parametric models. Assumption 4 is not hard to 
verify by using the fact that the normed quantity in (13.11) is bounded by D\ + D 2 , where 



D 1 = \\h(9,9') + I(9)\\, D 2 



1 k 

-Y J A(9,9',X, i )-h(9,9') 



i=l 
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with h(9,9') = E e [A(9,9',X)}. In fact, since h(9,9) = -1(9), provided that h(9, 9') is 
uniformly continuous, Di is uniformly small for \9 — 9'\ <C 1. On the other hand, exponential 
inequalities can be used for D 2 . For instance, if \A(9,9' , X)\ < M for some nonrandom 
M > for all 9 9' and X ~ fe, t nen Hoeffding's inequality gives F(D 2 > e) < A e~ Afc for 
some A , A > (|Pollardlll984h . As a concrete example, for the densities of N(9, 1), 9 G M, 



A(9,9',x) = 1 and hence Assumption 4 is satisfied. 

As 9 — 9 — > 0, the nulls become increasingly similar. By the asymptotic theory of 
statistics, to attain a fixed power while keeping the same significance level for the tests, k 
should grow at the same order as (9 — 9 )~ 2 . On the other hand, the results below deal with 
the case where k grows a little more slowly. First consider the univariate case d — 1. 

Theorem 3.1 Let Q a > 1. Denote 6 = 9 — 9q and k the number of replications per null. 
Suppose 

k = c\ ; such that s(5) — > oo and - ) — > as 8 — > 0. (3.2) 
d A s(d) ln(l/(r) 

Then, as 9 — ► 6*0, 



, , (l-o)o y^)^ / (lnQ Q ) 2 | 

Pw (a) = (1 + o(l))W 27r(1 _ Q)Q lnQa exp (-^J^ j • (3-3) 

Note that Q a > 1 if a + a < 1. In practice, since a is usually much less than 1 and a is 
small or only moderately large, the assumption Q a > 1 is not restrictive. 
The multivariate case d > 1 can be derived as a corollary. 

Corollary 3.1 Let Q a > 1. Denote 5 = 9 — 9 . Suppose k satisfies A3.2\) , with 5 2 being 
replaced by \S\ 2 . Let q(S) = S t I(9q)S. Then, as 9 — > 9 , 




, (l-a)a v / ^( 5 ) f ( ln <5a) i n 
PM (a) = (1 + °(l)W Ml _ a)a lnQa -p (-^y j> ■ (3-4) 



Under the condition of Theorem 13.11 it is not hard to see pk,s(at) — > 0. Therefore, the 
minimum number of nulls iV* and the minimum volume of data V* in order for (12.31) to be 
satisfied have the asympotics in (I2.6p . yielding 



1 27r(l- a ) a y^lnQ a f (lnQ a ) 2 

V* = fciv* = (1 + o(l))ln- W— t; r exp < — — . ro 

V K " l-j?y (l-a)a v^)5 P \2A;/(e )5 2 

On the other hand, if k has the same order as 5~ 2 , then by the Central Limit Theorem, 
in order to satisfy (12.31) . N* only needs to be a large constant and K is of the same order 
as 5~ 2 . To see in which case the minimum data volume is larger, it suffices to compare the 
orders of ^ exp {^} and 5~ 2 as 5 — > 0, where c > is a constant. By k = with 

s(5) — > oo, the ratio of the two is ^A/fcexpj^Ja} = e cs ^ / a/ s(5) — > oo. Therefore, when /c 
cannot grow as fast as 5~ 2 , a much larger volume of data is required. 
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3.2 Multiple tests on means of normal distributions 

Consider nulls Hi : 8{ = 6q for N(8i, cr 2 ), where a 2 is known and under false Hi, 9i = 9, with 
9 — 9 = 5 > 0. Without loss of generality, let #0 = and a — 1. Then f and f a are the 
densities of N(0, 1) and N(S, 1), respectively. By f a {x)/fo(x) = exp(5x — \b~ 2 ), the event in 
(12. 4p becomes 

lj=l 

Under true Hi, Ylj-^-ij ~ VkZ, while under false H i} Ylj Xij ~ VkZ + k8, where 
Z ~ JV(0, 1). Therefore, by (jO) . 

PM (a) = (1 _ „), ^_ + _ j + „, ^_ _ _ j , (3.5) 

where $(t) = 1 - = 1 - P(Z < t). 

Let Q Q > 1. If 5 | such that M 2 -> 0, then ^jfe ± ^ -> oo. Recall that, as i ->• oo, 

$(t) = $H) = (l + o(l))^4. (3.6) 

v Z7rr 



It is then not hard to get the asymptotic in Theorem 13.11 with fewer restrictions on (S, k) . 

Corollary 3.2 Let Q a > 1. Suppose kS 2 — > as 5 — > 0. Le£ /# 6e i/ie densities of N(8,<j 2 ), 
with a 2 being known. Then, as 9 — > #o ; 



, (l-a)a \pkb j o j (lnQ a ) 2 . 



2vr(l - a)a lnQ a { 2k5 2 /a 



u i 1 27r(l-a)ay^lng a J (hiQJ 2 \ 

= 111 T^V (l-a)a */* ^ i^V^j ' (3 - 8) 

The rapid increase of minimum data volume is illustrated in Figure I3T2T A). which graphs 
log 10 (Vt/V2) versus t G [0, 1] for 5 = 0.1, 0.2, and 0.4, where V t is the right hand side of (13.81) 
with k = <5~*. For the plot, a = 5%, a = 0.4 and p = 0.9. Even at the log scale, the increase 
in the minimum data value is apparent. 

3.3 Multiple tests on scales of Gamma distributions 

Denote by Gamma(a, b) the Gamma distribution with shape parameter a and scale parameter 
b. Multiple tests on the scales of Gamma distributions have been used as a case of study 
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in the literature ( Donoho fc Jin 20041 ). Fix v > 0. Let fo(x) be the density of Gamma(v, 1) 
and f a {x) that of Gamma(V, 1 + 5), where 5 > 0. Then 

lg— x ^.f— lg— 

M*) = ^) = ^) = r(,)(i + ^ ' x>0 - 

By f a (x)/fo(x) = (1 + <5)-"e fa /( 1+ *>, the event in becomes 

IsTy >> 1 [fc^ln(l + g) + lng a ](l + g) 
£<(A;) = I^Xy >c fc | , withc fc :=- - . (3.9) 

Under true ifj, X/j^y ~ Gamma(fcz/, 1); under false Hi, ~ Gamma(fcz/, 1 + 5). 

By (|2.7p . to get the asymptotics of iV* and K, the main step is to get the asymptotics of 
the probability of {S > c^} for S following Gamma(fcz/, 1) or Gamma(/cz/, 1 + 5). Since the 
tail probabilities under Gamma distributions are available in detail, the asymptotics can be 
attained for a much wider range of values of k than for the general case. The results are as 
follows; see Section IA.3I for a proof. 

Theorem 3.2 Let f Q and f a be as above and Q a > 1. Suppose 

(5, k) -> (0, oo) such that k5 -> oo, k5 2 -> 0. (3.10) 
Then, denoting ip(t) = t — y — ln(l + t) for t > —1, 



( s ( ( „ / (l-a)a y/ku5 J (lnQ a ) 2 f\nQ^\\ 
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3.4 Asymptotics of power 



For f i xed 5 and k, power can be analyzed using previous results (jSignorovitchl 120061 . IStorey 



20071 . IChill2008l ). Under the setup here, since (5, k) — ► (0, oo), the asymptotics of power are 



of interest. To avoid subtleties that a finite number of nulls may cause, we consider power 
under the situation where arbitrarily many nulls can be tested. For any procedure, let iV a 
and R a = R — R denote the numbers of false nulls and rejected false nulls, respectively. 
Then, provided the limit below exists, 

power^ = lim E[R a /N a ], (3.12) 

N—*oo 

characterizes the power of the procedure when N ^> iV*. We compare the powers of different 
procedures when they control the pFDR around or below the same level. For N ^> N*, the 
pFDR of a procedure can be characterized by 

pFDR^ = lim E[R /R \R>0). (3.13) 

As the limits in f)3.12p and 03.131) are defined for fixed S and k, we use power oc (5, k) and 
pFDR 00 (5, k) to indicate the dependency and consider the asymptotics of the quantities as 
(5, k) — > (0, oo). Fix a G (0, 1). First consider the thresholding procedure with cut-off a, 

d*(X) = 1 {P( 7?i = | X) < a} , i = 1, 2, . . . , 

i.e., d* rejects Hi if and only F(r)i = | X) < a. Denote by power^ (<5, k) and pFDR^ c (5, k) 
the asymptotic power and pFDR of d*, respectively. 

Proposition 3.1 Suppose (S,k) — * (0, oo) such that 

Pk,s{&) while staying positive, and (3-14) 
Pk,s{oii) = o{p kjS (a)) for any < a x < a, (3.15) 

Then d* has the following property 



for any fixed 5 > and k > 1, the limits in Ii3.12\) and Ii3.13\) exist, 
and R a /N a — > power^ as N — > oo. 



(3.16) 



Moreover, 



power^, k) = (1 + o(l)) (1 a)Pk ^ a \ pFDR^, k) = (1 + o(l))a. (3.17) 

(X 

We use d* as a baseline to compare other procedures that satisfy the basic property (13.161) 
while asymptotically controlling the pFDR. 
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Proposition 3.2 Let (S,k) — > (0, oo) as zn Proposition ^ .1\ Let d be a procedure satisfying 
A3.16\) with lim pFDR^ (5, fc) < at. // power^ (5, fc) > power^(5, fc) /or a// (5, k), then for 
any ol^ > a, 

power 00 (5, fc) < (1 + o(l))^M, pFDR (5, fc) = (1 + o(l))a. (3.18) 

a 

It is not hard to see that provided Q a > 1, the Pk,s(oi) given in Theorems 13.11 and 13.21 
satisfies ( 13. 14|) and (13.151) and therefore the above results apply. Since by Proposition 13. H 
Pk,s{ a 2) is of the same order as the power of a thresholding procedure with cut-off a 2) Propo- 
sition basically says that for any procedure satisfying (13.161) with limpFDR OC) (5, k) < a, 
its power is dominated up to a constant factor by a thresholding procedure with a cut-off 
just a little bit above a. In view of this, one question is whether there is a most power- 
ful procedure among those that satisfy (13.161) and lim pFDR^ (5, k) < a, and in particular, 
whether d* is such one. As seen next, in general the answer is no. Given c > 0, let d be a 
procedure such that for each (5, k), its cut-off is a + ck5 2 , i.e. 

di(X) = 1 {P(7/j = | X) < a + ckd 2 } . 

Proposition 3.3 Under the random effects model 112. let (5, k) be as in Theorem \3.1[ 
Suppose Q a > 1. Given M > 1, let c = (1 — a)aI(9o) In M/lnQ a . Then d satisfies $3.16}) 
and limpFDR 00 (5, k) < a, while power 00 (<5, k) = (M + o(l))power^ (<5, k). 

More generally, there are no asymptotically most powerful procedures that satisfy $3. 161) 
and limpFDR 00 (<5, k) < a. 

As an illustration, consider multiple testing for the mean values of N(9, 1) as in Section 
13.21 Figure I3T2T B) shows the dependency of the asymptotic power of d* on (5, k). Under the 
same parameters as in panel (A), it graphs log 10 P 4 , t G [1,2], where P t is (1 — a) /a times 
the right hand side of (13.71) with k = 5~ l . From Proposition 13.11 we know power^ c (5, k) = 
(1 + o(l) )P t as 5 — ► and k = 5~ f . The rapid decrease of power as 6 — > is clear. We 
next illustrate how the asymptotic power of a thresholding procedure can be arbitrarily 
increased by a small change in cut-off. As seen from (13.51) . for any 6 and k, power^ (5, k) = 

$ (j^f — an d the thresholding procedure with cut-off a + ck5 2 has power OC) (5, k) = 

l> ( lnQ ^± cM2 - . As (5, k) -> (0, 00) with k5 2 -> 0, the difference between the cut-offs a 

and a + ck5 2 tends to 0. It is not hard to get that for both procedures, pFDR OC) (5, k) — ► a. 
On the other hand, by (13.61) . the ratio of their asymptotic powers is 

) (\nQ a+ck6 2 Vk~5\ f\nQ a Vks\ 1 

(1+o(1))exp {(^s t) - \ju-—) J 

=(1 + Q (l))exp{ ( ' nQ °+^- ( '" g °> 2 } = (l + Q (l))exp{-g^} . 

Therefore, by increasing c, the power of the second thresholding procedure is arbitrarily 
many times higher than d*. 
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4 Summary and remarks 



This article studies the issues of minimum data volume and power when k = o(5~ ), i.e., the 
number of repeated measurements for each null is much smaller than the squared differences 
between false and true nulls. It shows that in this case, in order to meet a pFDR based 
performance criterion, the minimum data volume has to grow much faster than in the case 
where k is of the same order as 5~ 2 . It also shows how fast power will decay to and the 
sensitivity of the power to small changes in rejection rules. 

The results are essentially due to the fact that when k is not large enough, evidence 
against true nulls can only come from values of test statistics far away from the "normal" 
ones. When k increases more slowly than S~ 2 but faster than such values can be treated 
as moderate deviations (IDembo fc Zeitounilll998l ). which can yield the log-growth rate of the 
minimum data volume but nevertheless may not be accurate enough to give the growth rate 
itself. On the other hand, the article does not consider the case where k is only of the order 
of Analysis in this case seems to requir e the large deviations principle and can be quite 
subtle flChill20076l[Dembo fc Zeitounilll998h . 
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Appendix: technical details 

A.l Proof for the equivalence of criteria (11.11) and ( 12.31) 

We sketch a proof that under the random effects model ( 12. lj) . the criterion ( 11. II) can be 
rewritten as ( 12.31), where the infim um in ( 11.11) is taken over procedures satisfying ( 12.21) . For 
more details, see I Chi fc Tan! (120081 ). 

For any procedure d{X) as in (12. 2p . R = Yli=i di(X) and Ro = Y2i=i 1 {r]i = 0} di(X). 
Given X, if R > 0, then, as di(X) are now determined, 

i A 

E [R /R | X] = - V di(X)Pfo = | X) > minPfa = | X), 

ix ' i=l 

i=l 

with equality if and only if d only rejects Hi with the smallest P(t/j = | X). On the other 
hand, if R = 0, then by definition, E[Rq/R \ R > 0, X] — 1. Note that by Bayes rule, 



pfa = o\x) = F( Vi = o | x i:j , j = i, . . . , k) 



1 



n 



fa{Xi 



1 - a jJl fo( x i 

It is then seen that the criterion (11.11) can be rewritten as (12. 3p . 



(A.l) 
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A. 2 Proofs for general multiple tests 

Recall that by Bikjalis' theorem f Nagaevl 1979 ). there is an absolute constant f3 > 0, such 
that for any Zi, Z 2 , . . . i.i.d. with EZ\ = 0, Var(Zi) = a 2 > and Pl-Z^ 3 < oo, 



< 



f ^ , *=1,2, 

(7 3 ^(1 + |t| 3 ) 



(A.2) 



Proo/ of Theorem\3Ji For G 6 C R, let 5 = 9 - 9 . Then 

A(e ,e, x )s 2 



x)-l%,x) = £(9 ,x)5 + 



x)S — 



A(9,9 ,x)S 2 



(A.3) 
(A.4) 



(A.5) 



According to (I2.6p . we need to compute Pk,s(oc). By (12. 7p . 

Pk,s( a ) = (! _ a)P9 {Ek) + aP e (E k ), 

where P# is the fc-fold product of the probability measure with density fg and E k is the event 
...,X k ): Ei=M^ x i) -^ ,Xi)] > \nQ a }. Under f 0o , £(0 o ,*i) are i.i.d. with mean 
and variance I(9 ). Given e > 0, define events 

k 

l^2A(9 Q ,9,X i ) + I(9 ) 



i=l 



<e\. 



Denote Z { = £(9 , By (TOl) . F_ D C H G/i C F + fl G fc , where, for ^ 0„, 



sign(5) 



± 



> u± > with u-t 



lnQ Q + ffc(/(0 o ) Te )<5 2 



Without loss of generality, we only consider the case 9 > 9q. Then 

P 6o (F_) - P 9o (G c k ) < P 6o (E k ) < P 6o (F + ) + Pg Q {G c k ) 
By (IA.2I) and Assumption [31 



(A.6) 



|$(« ± )-P 0O (F ± )| < 



/3M 3 



>/fc/(0 o ) 3/2 (i + |u±| 



We need the following results. 

Lemma A.l If k satisfies [3.2$ . then, as 5 — > ; 

$(u±) = (l + o(l)) 
1 



u|/2 



V^(l + 



27TU± 

o($(u ± )), 



P, (G c fc )=o($( M± )). 



(A.7) 

(A.8) 
(A.9) 
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Assume Lemma [A. II is true for now. By (1A.6I) - (1A.9|) . there is r k — > 0, such that 



-v?_/2 



27TU- 



<Pe (E k ) <{l+r k ) 



-u\/2 



Since k5 2 — > 0, U- ~ u + ~ . ln = . On the other hand, 

y/kI(6o)S 



(lnQ Q ) 2 , (I(9 )Te)\nQ a , (/(0 o )+e) 2 M 2 



+ 



+ 



± fc/(fl )<5 2 ' /(0 O ) 4/(0 o ) 

By Assumption [TJ /(#o) > 0. Since e > is arbitrary and A;<5 2 — > 0, we then get 



P* (P*) = (l + o(l)) 



'27TlnQ a * I 2kl(9 )5 2 
With similar argument, now applied to £(8,Xi) under /g, 

, , . , JkJW)5 f (lnQ Q ) 2 



(A.10) 



ln Q Q 



where +|lnQ Q in the exponential is due to — \A{9, , x)8 2 in ( 1A.4I) . By Assumption EJ 
there are constants C > and r > 0, such that for \S\ < r, 



kl(9)5 2 kl(6 )5 2 



< 



C 



k\5\P{6 Q ) 



Since kS — > oo, it follows that 



Pe(-Bjfc) = (l + o(l)) v exp <^ - + 

V27rlnQ Q [ 2kl(9 )0 2 

Combining (jAT5j) . flA+UD . flA~TT|) . and exp{^} = 



lnQ c 



(l-q)(l-q) 



(13. 3p then follows. 



(A.ll) 



□ 



Proof of Lemma \A.l\ Because u± —>■ oo as 5 — > 0, (IA.7[) follows from (I3.6p . To show (]A.8j) . 
it suffices to show \fku 2 ± e^ u± ^ 2 — > oo as <5 — > 0, or, equivalently, ln — tt^ + 4 ln w± — > oo. 
Because u± is of the same order as and k5 2 — > 0, it is seen the above asymptotic follows 

if = o(ln/c), or (Hn&;)<5 2 — >■ oo. Now by s(8) — > oo and s(<5) = o(ln(l/5)), 

(A; In fc)5 2 = — ^— ( 2 In i - In s(5) ] <5 2 -> oo. 
o / s(o) \ J 



To show flA.9j) . let A > be as in (|3.1|) . Then P^G^) is of the same order as e" Afe . By 
(IA.7j) . it suffices to show u 2 ± = o{k). Since u 2 ± is of the same order as l/{k5 2 ) and k5 — > oo, 
the last claim is proved. □ 
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Proof of Corollary \3.1\ In place of flA.31) and flA.4j) . we have 

£(9, x) - i(9 , x) = £(9 , xfS + 6TA ^ X)S = m x) T 8 ~ 6TA ^ X)6 . 

Let e = 5/\5\. Under f do , £(9 , X^) T e are i.i.d. with mean and variance ^(6*0) = e T I(9 )e > 0; 
under f 9 , £(9,Xi) T e are i.i.d. with mean and variance v{9) = e T I(9)e. Applying the proof 
of Theorem O to £(9 , Xife and £{9, X^e yields 



a)a \fkv(9o) \8\ ( (\nQ c 
PkA a ) ~ \l WT, \ ex P 



_ 2tt(1 -a)a lnQ Q ^ [ 2kv(9 ) \5\< 
Since ^(6> ) |^| 2 = S T I(9 )5, flUD then follows. □ 



A. 3 Proofs for multiple tests on Gamma distributions 

We next prove Theorem 13.21 Denote by G k (x) the upper tail probability of Gamma(b, 1), 
i.e. Gk(x) = s fciy_1 e~ s ds, x > 0. As noted in Section I3~3"l the main step is to find 

the asymptotics of Gk{ck) and G k (d k ), where c k is defined in (13.91) and d k = jjh. 

To find the asymptotic of G k (c k ), first, by power expansion of ln(l + 5), for 5 G (—1, 1), 

Ck = kv + — + ku6^ {j + + 2) + lnQ a , 
d k = ku-\ h kvb > — : — - — . 

6 U 3 + 2 

Because k8 — > 00 while k5 2 — ^ 0, it is seen that in each of the sums, every term is of an 
infinitesimal order of its previous one. Let 

z = ku, s = z(l + t), b = lnQ Q . (A.12) 

With the variable substitutions, 

G k (c k ) = Z -^— / (1 + ty- l e~ zt dt, G k (d k ) = / (1 + t) z - l e- zt dt, 



T(z) .lb, nfx ^ Viz 



V v ' ^ 



where 



I(z,S) J(z,5) 



D(6, z) = 5r(5) + ± with r(5) = ^ g) , (A.13) 

L(5, z) = Sr(6) + -, with f(S) = - V fc^l. (A.14) 

2; 7 + 2 

i=o J ^ 
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The main step is to show 



I(z, 8) ~ - exp 
6 



b 2 b f b 



JM)~-exp + 



Assume the two formulas are true for now. By Stirling's formula, z r e ^ 
Then by §KjB) and fTATTBl . 



(A.15) 
(A.16) 

( 1 + °( 1 ))v / 5- 



Gk{dk) 





r 6 2 








[ 2z5 2 



+ - — zip 



b 

Is 



Since ^,5(0) = (1 — a)Gk{ck) + aGk{dk) and (IA. 12[) 



PkAa) 



(l-a)a / (lnQ a ) 2 /^hiQ a 



27r(l — a)a; ^/ln Q c 



2ku5 2 



kvb 



The proof is complete by (12. 6p . 

The rest of the section is devoted to the proof of (1A.15I) and (1A.16I) . Observe r(5) — > 1/2 
and f(5) -> -1/2 as 5 -> 0. By (I3TTUD . 



z5 = vkb — > 00, 2;5 2 = vkb 2 — > 0. 



(A.17) 



It is then not hard to check that D(5, z) ~ 5/2. Also, for any z > 0, (1 + 1) 2 x e zt is strictly 
decreasing in t > 0. Given < e < 1, using flA. 17h again, 



Ie(z,S) :-- 



(i + ty- L e- zt dt> / (1 + ty- L e- zt dt 



H 1+ 

(a) 

> exp 



36 

15 



z-l 



-36/(5 



z5 



36 1 /36 
~b ~ 2 \z6 



, , N 36 6 f C 1 6 



for some C > 0, where (a) is due to ln(l + x) > x — : x 2 /2 for x > 0. Therefore, J e (z, <5) 1/z 1. 
On the other hand, 



(/•oo \ l/z 

J (l + *)*- 1 e-**d*J ^sup(l 



t)e -t < 1. 
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As a result, for any e > 0, I(z,5) ~ I e (z,5). Since e is arbitrary, it follows that we can 
replace (1 + t) 2 " 1 in the integrand to (1 + t) z to get 

I(z,S)~ f (l + t) z e- zt dt= [ e~ Mt) dt, (A. 18) 

J±+D(S,z) J±+D{5,z) 

where ip(t) = t - ln(l +t). By ip'(t) = ^ <p(t) is a strictly increasing function from (0, oo) 
onto (0, oo) with smooth inverse ip~ l {u). On the other hand, (p(t) = y + C(t 3 ), as t | 0. As 
a result, as u — > 0+, <^ _1 (it) = (1 + o(l))v / 2u and hence 

= 1 = i + _J_ = L+gffl as u — ► 0. 

By (1A. 18[) and the arbitrariness of e > as well as the above properties of ip, 

r<p(e) e~ zu f°° e~ zu 
I{z,8) ~ / du ~ / du = I\. 

J<p(±+n(8,z)) V2u J <p(±+D(8,z)) V2u 

By variable substitution u = v/z, 

1 Z" 00 1 
h = — \ ^ e ~ V dv - 

Since z<iD(<5, z) — ► oo and D(5, z) — > 0, by ~ t 2 /2 as i — > 0, 

„£(.»)' = JL^oo. (A.19) 

Recall that for any a, t a e~ l dt ~ x a e~ x , as x — > oo. Then by ( 1A.19I) 

Ji ~ -i= 1 =exp(-2y (A + D(5,z] 

2z<f(-+D(5,z] 



| exp j — z(p ( A + -D(5, 2) 



z<5 



Because ^(t) = — 



2 ' 



^(- + J D(^)I =^(^ + D(6,z)\ +zip\- + D(6,z: 



First, by (jA"17jl . 



2U + ( ' V = 2^ + ^^ + 2 

ISr(8) + -)+-(8r(8) + -) = _ + - + (i). 



2z5 2 <5 V w z 7 2 V w z / 2z<5 2 2 
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Second, since tp'(x) = (f'(x) — x = 1 — — x = —j^,, by Taylor expansion and (IA.13I) 

Ts + m z) ) - * (s) = zD{S < zW {ts + ™ z) ) = T (s) 2 = «»' 



^ +D(w )=^ + » + *(» l +0 (l) 



£■0 

As a result 



and hence by I(z, 5) ~ I\, flA. 15[) then follows. By similar argument, it can be shown that 



■ /( 1.6) ~ ^_ 1 = exp j-z</? + L ( 5 > ^ 



'2z<p (±+L(5,z)) 



5 f r/f \ 

exp < — Z<£> I — + L(0, Zj 



6 "1 r V 25 
which leads to ( 1A.16I) . 

A. 4 Proofs for the asymptotics of power 

A basic fact to use is that under the random effects model (12.11) . P(r/j = | X) are i.i.d. and 
by (lA.ip . given 5 > and k, for any a G (0, 1), the probability of {P(?7i = | X) < a} is 

p M (a) = (1 - a)P {E k {a)) + aP a {E k {a)), (A.20) 

where Pq and P a are the probability distributions under true and false nulls, respectively, 
and E k {a) = {n - =1 > Qa} with X u ...,X k being i.i.d. 

Proof of Proposition \3.1[ For fixed N, 

N N N 

R=Y,d*(X), R = Y i dl{X)(l-TH), R a = ^2dl(X)r h . 

i=l i=l i=l 

Given 5 > and k, since d*(X) = 1 {Pfo = | X) < a}, by (|A~20|) and the Weak Law 
of Large Numbers (WLLN), R/N ^ p k ,s( a ) > °- Similarly, R /N ^ (1 - a)P (£ fc (a)), 

p 

R a /N — > aP a (E k (a)) and N a /N — ► a. Property (13.161) can then be proved. In particular, 

power 00 (5, fc) = P.(S fc (a)), pFDR^, fc) = (j^W^O) , (A . 21) 
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To show (I3.17p . given X with R > 0, 



1 - 

E [Rq/R \X] = -J2E[1 {f(Vi = 0\X)<a}(l-rn)\ X] 



R , 

i=i 

N 



= ^ E 1 = I X) < a} P(7ft = | X) 

8=1 
1 ^ 

<-]>>! {P(^ = 0|X)<a} 



a. 



i=l 



Since Pk,s((x) > 0, P(g?*(X) > for at least one i — 1, . . . , iV) > 0. Therefore, the conditional 
expectation of Rq/R over X with R > is well defined, giving E[R /R \ R > 0] < a. Thus 
pFDRj! X) (5, £;) < a for all (<5, k). On the other hand, given ft < a, 

1 - 

E[Ro/R\X) > -J2E[l{(3<n^ = 0\X)<a}(l-r ]t )\X] 

i=l 

8 N 

>^l{/3<P(^ = 0|X)<a}. 

i=l 

Under the random effects model and the WLLN, 

N 

1 {ft < HVi = | X) < a} = (1 + Op(l))[p M (a) - p M (/W 

p 

where o p (l) stands for some sequence of random variables £tv ~^ as N — > oo. Taking 
expectation over X with i? > and then letting iV — > oo, 

P FDR^(5,A;)>^mM^mM. 

Let (5,fc) -> (0,oo) while satisfying (jSHj) and (T37T5]) . Then ImTpFDR^(5, fc) > /?. Since /? 
is arbitrary, pFDR^ c (5, k) — > a, showing the second half of ( 13. 17ft . Finally, combining this 
with fjX20j) and ([Oil) , the first half of (13~TTD follows. □ 



Proof of Proposition lcTB . Let d be a procedure more powerful than d* while satisfying ( 13.161) 
and limpFDR oc (5, k) < a. Let (S, k) be fixed first. Given < a\ < a < a 2 , let 

rW = #{ z : F( Vi = I X) < a 1; = 1}, 

flP) = #{z : ai < = | X) < a 2 , = 1}, 

#(3) = ; F(rji = I X) > a 3 , di(X) = 1}. 
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Then R = RW + R^ + and for any X with R > 0, 



1 A aiRW+aoR® 
E [Ro/R \X) = -J2 di&MVi = | X) > -± ±-* . (A.22) 

8=1 

Since d satisfies (13.161) . by the WLLN, R a = (a + o p (l))power 00 (5, k)N as N — > oo. By 
the assumption and (13.171) . power oc (5, k) > power^(<$, k) = (1 — a + o(l))p^$(a) / a. Since 
R > Ra, R is at least of the same order as pk,s{ot)N. On the other hand, since 

rW < #{ t : pfo = o | X) < ai} = (1 + o p (l))p fcj5 (a 1 )iV, 
by (l3~T5l) . = o p (l)i2 and so # (2) + R {3) = (1 + o p (l))i2. Therefore, 



— = (1 + o p (l)) p . 2 . ■ p ,„. = (1 + o p (l)) 



«i + (a 2 — oil) 



Combine this with ()A.22j) . Taking expectation over X and letting N — > oo yield 
pFDR DO (5, fe) > «i+ (a 2 -ai) lim E[R {2,) /R \ R > 0]. 



Let (5, fc) — > (0, oo). As limpFDR 00 (5, k) < a by the assumption and «i < a is 
arbitrary, the second half of (I3.18P follows. Furthermore, the above inequality implies 
iimiv E[R®/R | R > 0] = o(l). Since for each fixed (5, fc), itf 1 ) = o p (l).R and N a = 
(a + o p (l))A^, it then follows that, as (5, k) -> (0,oo), limjvS[i2/JV a ] = limjy S[(-R (2) + 
i?( 3 ))/iV a ] = (1 + o(l)) limjy -E[-R (2 V^"a] • Since i?( 2 ) is no greater than the number of nulls 
with F(r]i = | X) < a 2 , which is (1 + o p (l))p k)S (a 2 )N, 



power OC) ((5, fc) < lim _E 

AT— >oo 



R 



[l + o(l)) Hm £ 

JV— >oo 



RW 

N a 



< (l + o(l)) 



Pfc,j(a2) 



Therefore, power oc (5, k) satisfies (13.181) . 



□ 



Proof of Proposition 13.31 Denote a k = a + ckS 2 . By the WLLN, d satisfies ( I3.16p . For 

each (5, k), d is a thresholding procedure, so pFDR DO (5, fc) < Then by kS 2 — > 0, d satisfies 
lim pFDR^ (S, k) < a. Given (5, k), following the same argument that leads to (lA.llj) . 



^/kI(6oj5 / (In Q Q J 2 lnQ Qfc 



power oc (5, fc) = (1 + o(l)) exp <^ - \ + 

V2vrlnQ Qfc I 2kI(9 )o 

Since <5a fe — > Q a , to get power oc (5, k) = (M + o(l))power^(5, k) as (5, fc) — > (0, oo), it boils 
down to showing 

/ (lnQ a J 2 | / (lnQ c 

ex P \ ~ r,7 r//i Nf9 r = (1 + o(l))M exp J 



2kl(6 )5 2 J v ^ [ 2kl(6 )5 2 
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By Taylor expansion, 



InQ 



InQ, 



1 ~ Qfc 
1 — a 




ck6 2 



+ 0{{k5 2 ) 2 ). 



a 



111 



(1 — a)a 



As a result, 



(lnQ Qfc ) 2 (lnQ Q ) 2 
2kl(6 )5 2 2kl(6 )5 2 



(1 - a)al(9 ) 



cln Q, 



+ 0(fc<5 2 ). 



By the definition of c, the result follows. 

Finally, we show that among procedures that satisfy (13.161) and lim pFDR^ (5, k) < a, 
no one is asymptotically the most powerful. It suffices to show that for any such procedure 
d that is more powerful than d*, there is yet another one more powerful than d. First, by 
diagonal argument and the first part of ( 13.181) . there is a decreasing — > a, such that 
power oc (5, k) < 2pk t s(ak)/a for large k. Now we use the same construction as above. Let 
oc' k = ak + Ckk5 2 , with cj~ = (1 — ak)akI(0 o )M/ \nQ ak , where M > 0. The thresholding 
procedure using a' k as cut-offs satisfy the conditions of Proposition 13. 21 It is seen that as 
long as M is large enough, the power of this new procedure will be greater than Ap^sioc^/oc, 
and hence at least twice as large as power oc (5, k). □ 
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